The question comes as the general secretary of ruling LDP political party, Toshihiro Nikai, raised the possibility the day before.
COVID-19 cases have been surging across Japan.
The U.S. State Department suggested an Olympic boycott was among the possibilities but a senior official said later a boycott has not been discussed.
Athletes will not be required to be vaccinated to enter Japan, but many will be.
Having no fans will be costly.
Japan has attributed about 3,800 deaths to COVID-19 in a country of 126 million.
Asked how clean he thinks the Games will be the 23-time Olympic gold medalist bluntly told CNN Sport's Don Riddell: "Four or five" out of 10.
Although the length of the 121-day torch relay will not be shortened, the number of staff present and use of official vehicles will be reviewed.
The IOC and organizers have also said the Olympics will not be postponed again and would be canceled.
The Olympics could be canceled if the Covid-19 pandemic continues into next year, according to the Tokyo 2020 president.