Just leave out the "him". "Work with" isn't used as a verb and preposition here. It's used as part of the descriptive phrase "said to have been difficult to work with" that follows the main verb "is". (I believe this phrase is called a "predicate adjective" or a "subject complement.") "Work with" can be used with verbs other than "to be" or on its own, and then takes an object. I want to work with him. Please work with him.