Each trunk on which you wish to have certain callers routed to the Auto Attendant must have an appropriate InboundCallRoute rule:
InboundCallRoute : {12341234567:aa},{ph}
or
InboundCallRoute : {(12341234567|12347654321):aa},{ph}
where 12341234567 and 12347654321 are CallerID's to be routed to the Auto Attendant.
The Auto Attendant PrimaryLine setting is only used for outbound calls made through the Auto Attendant.