Someone asked this in the comments of a very only post and I can't see their email address so hopefully they see this answer.
I know the cause but I don't know why. The list of contacts is provided by a Data View web part which contains the following...
<ParameterBinding Name="SearchQS" Location="QueryString(Search)" DefaultValue="r"/>
Editing the page with SharePoint Designer and changing the default value to something like ! will result in no contacts being displayed when the page is first loaded.