An image signal processing system includes a buffer memory for storing a re-sampled digital data and a memory controller for receiving an used memory depth value and controlling the read speed of the buffer memory for the re-sampled digital data according to the used memory depth value, thereby solving the overflow or underflow problems of the memory.