You need to learn about the different types of PDFs. PDF/ A is an archive format that does not allow changes. PDF/X is used for graphics exchange
For forms I would use only the PDF with a target version. Do not use the Optimizer feature.
Forms always become larger due to the need to include the font for the various form fields. You might have to review the type fonts used by the form fields and one standard font as much a possible.
It is possible to use the FDF file to move field values from one PDF to another.
If you are trying to replace the underlying content and keep the form fields, use the replace pages.