Kurt Watson will serve as the school's interim athletic director.
The decision comes just days after Jayhawks football coach Les Miles left the program.
"Jeff guided Kansas athletics to progress in student-athlete healthcare, diversity and inclusion, and student-athlete academic achievement, all while managing significant challenges not of his own making," said Kansas chancellor Douglas Girod. "Most important, Jeff was unwaveringly dedicated to students, coaches, and staff, and he represented KU with integrity and compassion. For that, we thank him, and we wish him the very best."
Long began his tenure at Kansas shortly before hiring Miles as the school's football coach in 2018. The two had a relationship dating back to their time working together at Michigan during the late 1980s and early '90s.
Miles was placed on administrative leave and subsequently left the Jayhawks' program following accusations of inappropriate behavior toward female students during his tenure at LSU.
New offensive coordinator Mike DeBord was named acting head coach.
In a press conference Tuesday, Long said he and the university were unaware of Miles' past wrongdoings before hiring him.
"I also asked coach Miles directly during the interview process whether there was anything in his past that could potentially embarrass the university or himself or our program, and he said no. We also did our due diligence by talking to individuals within the LSU athletics department to see if there was anything we should be aware of regarding coach Miles' tenure at LSU and received no indications of any issues," Long said, per Pete Thamel of Yahoo Sports.
Long was expected to lead the search for the Jayhawks' next football coach, but that task will fall to his successor.
Long's departure comes as the Kansas men's basketball team awaits a decision on NCAA sanctions regarding five Level I violations.