I just made a patch for Comix 4.0.4 to add this feature. I even added a nifty slider to the dialog.
It works for me. Feel free to play with it, I'd love to see this feature in the next release.
-Added new PageSelector class that handles the "Go to page..." dialog
-Added "Go to page..." menu item to the Go menu
-Added "Go to page..." button to the toolbar
Changed keyboard shortcuts/hotkeys (now these are the same as Comix v3.x.x):
-"G" hotkey now opens the "Go to page..." dialog, instead of the magnifying glass
-"Z" hotkey now toggles the magnifying glass on/off
-Replaced shortcut "G" (which is now "Go to page") with "L" in the _View->Magnifying G_lass menu item
(in Comix version 3.x.x it was Magnifying _Lens, hence the "L")