dragging on a gameObject messes with collisions

I have a player with movement that works with collisions as long as I don't click directly on the player before dragging. If I click anywhere else, everything works fine.

I'm not sure what I'm missing and days of research hasn't resulted in anything. Am I missing something super basic? Each collider has a rigidbody, appropriate settings, and (obviously) colliders. It's something to do with clicking directly on the gameObject before dragging.

Here's my code:

using System.Collections; using System.Collections.Generic; using UnityEngine; public class playercontroller : MonoBehaviour { Rigidbody2D rb2d; private Vector3 mouseposition; private Vector2 direction; private float movespeed = 100f; // Start is called before the first frame update void Start() { rb2d = GetComponent(); } void FixedUpdate() { if (Input.GetMouseButton(0)) { followmouse(); } } void followmouse() { mouseposition = Camera.main.ScreenToWorldPoint(Input.mousePosition); direction = (mouseposition - new Vector3(transform.position.x, transform.position.y-1, transform.position.z)).normalized; rb2d.velocity = new Vector2(direction.x * movespeed, direction.y * movespeed); } }