Scottish referees claim the most recent examples of the "ongoing trend of personal and unjust criticism" towards top officials is "unacceptable."
The comments were made by Crawford Allan, Scottish Senior Football Referees Association chairman.
Inverness manager John Hughes also criticised referee Andrew Dallas.
Neilson disputed Collum's decision to send off Callum Paterson for a challenge on Darian MacKinnon with the visitors leading 2-1.
After the 3-2 loss he said: "This game, with the officials we were getting, we knew there was a high probability of a man being sent off and it's happened again."
After Inverness's draw at Dundee, Hughes said Dallas "brought a nervousness to the game which didn't help" and wasn't "ready" to referee top flight football.
Scottish Football Association compliance officer Tony McGlennan is to review the comments.
On Monday, SFA chief executive Stewart Regan defended the match officials, saying: "Our referees are operating at a very high standard, we are punching above our weight. It's just unfortunate referees are sometimes whipping boys for other performances on the field."
The referees' association welcomed Regan's intervention in its statement, which began: "The Scottish Senior Football Referees Association (SSFRA) is compelled to express its disappointment at the ongoing trend of personal and unjust criticism towards our most senior match officials.
"The SSFRA is committed to supporting their members and colleagues, whilst at the same time safeguarding the recruitment and development of future referees.
"We believe the nature of comments made this weekend serve only to make the process of retention and recruitment more difficult - to the detriment of the whole game.
"Referees accept that decisions made during a match will be reviewed and scrutinised. However, the extent to which this criticism has become focused on individual match officials is a concern to the SSFRA, as is the unhealthy and unbalanced focus on refereeing decisions in media outlets.
"We believe the behaviour which manifested itself again at the weekend to be unacceptable but given these instances may be subject to judicial panel proceedings, it would be inappropriate to comment further at this stage."
The SSFRA added that it hoped "to achieve a better mutual understanding and working relationship for all" by discussing matters with PFA Scotland and the Coaches and Managers' Association at the SFA Congress.