Having a slow or fast mobile site should not impact your mobile ranking, assuming your desktop site is fast. Google still uses desktop signals for your mobile rankings. Google's mobile-friendly algorithm is launching soon but Google's Gary lllyes says factors like page speed signals for ranking are based on the desktop version, not the mobile version.
