At least 18 people were injured in a residential area. Two of them, an 11-year-old girl and a 39-year-old man, have died.
A suspect, a man in his 50s, stabbed himself in the neck after his rampage and later died in hospital.
Violent crime is rare in Japan and the motive for the attack is unknown.
The suspect was holding knives in both hands as he attacked the victims – 16 of whom were schoolgirls.
Police later named the victims of the fatal attack as Hanako Kuribayashi, who was in the sixth grade, and foreign ministry official Satoshi Oyama, who is believed to be the parent of another child.
The suspect has not been named.
Japanese Prime Minister Shinzo Abe condemned the attack, saying he felt “strong anger” against it.
“It is a very harrowing case. I offer my heartfelt condolences to the victims and hope the injured recover quickly,” Mr Abe said.
The incident took place just before 08:00 local time on Tuesday (23:00 GMT Monday) near a local park and a train station.
A number of students were waiting to board their bus to the nearby Caritas elementary school, a private Catholic school.
The driver of the Caritas bus told NHK that he saw a man approach the queue of students.
He began stabbing the children and then boarded the bus and stabbed children inside as well, according to the BBC’s Japan correspondent, Rupert Wingfield-Hayes.
News agency Kyodo said the man was heard shouting “I’m going to kill you” during the attack.
He was apprehended by police and later died in hospital from injuries.
Locals have now begun placing flowers outside the site of the attack in a tribute to the victims.
People have begun laying down flowers at the site of the mass stabbing
US President Donald Trump, who was in Japan on the last day of a state visit, also offered his “prayers and sympathy” to the victims.
How common are such incidents in Japan?
They are extremely rare and Japan has one of the world’s lowest rates of violent crime.
Our correspondent says that in his neighbourhood it is very common to see children as young as four walking to school by themselves – a normal practice in Japan because it is so safe.
However, there has been in recent years a handful of attacks involving knives.
In 2016, 19 people at a care centre for people with mental disabilities were stabbed to death by a former worker. He reportedly said he wanted those with disabilities to “disappear”.
Eight students were killed in a 2001 attack after a man entered an elementary school in Osaka.