Climate Change and Revenue Minister Simon Watts will take Lee's spot in Cabinet.
Invercargill MP Simmonds was a minister outside of Cabinet.
Prime Minister Christopher Luxon announced the changes this afternoon, five months to the day after he first announced the ministerial roles and responsibilities.
Justice Minister Paul Goldsmith - currently overseas - would take over the Media and Broadcasting role, while Social Development Minister Louise Upston would pick up Disability Issues, he said.
Lee was under pressure after Warner Brothers Discovery announced it would stop producing local news through Newshub, and shutter the majority of its operations in New Zealand.
She faced repeated questions about what the government would do about the closure of Newshub, and Labour said she had "more than enough time" to find solutions.
Simmonds had also been in headlines over the handling of changes to disability-related funding. She admitted the handling of the disability funding changes - which included restricting the way equipment and support services were funded - was bungled, and later apologised for it.
She signed off on the decision a few days before it was announced on the ministry's social media accounts, taking disabled people and carers by surprise.
Labour said the changes were callous and a broken promise, and leader Chris Hipkins called for her to be sacked over it.
After the changes, Finance Minister Nicola Willis said future decisions on the funding would have to be taken to Cabinet.
'Changing circumstances'
Speaking to reporters today, Christopher Luxon said the changes were about making sure the government had "the right people on the right assignment at the right time".
"In both these cases of both these portfolios there have been significant changes and complexities added to them over the course since the ministers were allocated these responsibilities ... there's a lot more complexity added to these portfolios."
He avoided saying whether either of the ministers had done anything wrong, despite multiple questions about why they deserved the demotions - particularly Lee, who had been an MP for 16 years and held the media portfolio for the National Party since 2017.
Lee's removal from Cabinet was a "recognition that there is a lower workload" and did not mean she would not return to Cabinet at a later date, he said, but changes in the media industry had "moved quicker, faster, sooner and as a result I want to make sure that there is a good senior Cabinet minister responsibility around the issues".
On disability issues, he said there had been "a habit now" of cost overruns and poor financial management, but there was "innately more complexity" in both portfolios.
When asked if it was a warning shot to his caucus, he said he was just a person who "will adapt very quickly and dynamically to changing circumstances and situations".
"This is how I roll, this is how I lead ... I appreciate this may not be the way things have been done in the past here, but expect this to happen going forward as well."
He had spoken to the relevant ministers about the decision earlier this morning, and it had been a "tough day" for them, he said.
"It's understandable ... it is disappointing if you're the individual, but the reality is they know that they are really valued by our team, we have full confidence in them, they're doing a good job on their other portfolios and they have important contributions to make."
Luxon said he had informed both his coalition partners, ACT and NZ First. Asked if he would have the authority to use the same approach with them, he said: "I'm the Prime Minister and I determine ultimately the performance of my Cabinet ministers".
He said they had a "very strong cadre" of women at the heart of the government doing good jobs.
In an earlier statement, Luxon said it had "become clear in recent months that there are significant challenges in the media sector. Similarly, we have discovered major financial issues with programmes run by the Ministry of Disabled People".
"I have come to the view it is important to have senior Cabinet Ministers considering these issues."
He said there were "significant synergies" between Goldsmith's Arts, Culture and Heritage portfolio and the media role he would be taking up.
He said he had asked Upston to pick up the disability role because Whaikaha, the Ministry of Disabled People, was a departmental agency within the Ministry of Social Development.
"This will free Penny Simmonds up to focus on the Environment portfolio and the major changes she is progressing to improve tertiary education," he said.
Lee retains her Economic Development, Ethnic Communities and Associate ACC roles as a Minister outside Cabinet.
Simmonds, who remains outside Cabinet, retains Environment, Tertiary Education and Skills, and Associate Social Development and Employment.
Wheels falling off already, says Hipkins
Labour leader Chris Hipkins says the demotion of two ministers shows Luxon "didn't have particularly good judgement".
Lee and Simmonds were "never up to the task that they were given", he told media today, and for Luxon to say things had dramatically changed in the media landscape showed he had not been paying attention before he became Prime Minister.
The government had only been in office for six months, and "the wheels are falling off already", he said.
"New Zealanders could see very clearly several months ago that Penny Simmonds was making an absolute hash of the disability issues portfolio. She'd gotten off on the wrong foot, had completely messed things up, and it's taken [Luxon] months to actually do something about that."
He believed Simmonds handled the disability issues portfolio "abysmally" and the ways she has portrayed the community have been "reprehensible".
"The idea that people who are living with a disability and those who care for them are somehow ripping off the system - which is a claim that she's repeatedly made - is just wrong.
"But not only is it wrong, it's incredibly hurtful."
He said Melissa Lee's "invisibility" during a crisis in the New Zealand media has been "baffling", but it was hard to say what's been going on with her because the government "haven't been saying anything publicly about the onslaught that the New Zealand media has been facing".
"Melissa Lee tells us that she was working on a proposal that was ultimately vetoed by the government's coalition partners and appears to have lost her job for something that [NZ First's] Winston Peters was ultimately responsible for ... although that's guesswork because the government hasn't been upfront."
Upston had made clear her number one priority was to get people off benefits, Hipkins said, and "I think those in the disability community will be feeling uncomfortable about that".
He believed Luxon should be asking "why stop there", and turn his attention to ACT leader David Seymour and NZ First Minister Shane Jones, who both criticised the Waitangi Tribunal, which he argued was criticism of the judiciary.
"He publicly reprimanded them, and then David Seymour reprimanded him back in return. I think that shows that his own ministers don't have any respect for his authority."
He referred to Stuart Nash's resignation from Hipkins' government after publicly criticising a judge's decision, saying "he resigned the same day it happened" and he thought the issues was "not something that [Luxon]'s going to take any action on".
