That depends on the rail traffic going through. Sometimes it's busy on a Saturday. Sometimes it's not.
As a minimum you should time your visit to correspond with the arrival of the VIA trains from Ottawa and Brockville. This results in a nice passing of the two trains. And, if there are a couple of freights changing crews, you have an added bonus.
I go to the Smiths Falls Railway Museum very regularly and I always pass through the CP Rail/VIA tracks to see what is there.
And you can always visit the Museum.
Make sure you bring your camera or camcorder.